Poor Metacomprehension Accuracy as a Result of Inappropriate Cue Use

Two studies attempt to determine the causes of poor metacomprehension accuracy, and then, in turn, to identify interventions that circumvent these difficulties to support effective comprehension monitoring performance. The first study explored the cues that both at-risk and typical college readers use as a basis for their metacomprehension judgments in the context of a delayed summarization paradigm. Improvement was seen in all readers, but at-risk readers did not reach the same level of metacomprehension accuracy as a sample of typical college readers. Further, while few readers reported using comprehension-related cues, more at-risk readers reported using surface-related cues as the basis for their judgments. To support the use of more predictive cues among the at-risk readers, a second study employed a concept map intervention, which was intended to make situation model-level representations more salient. Concept mapping improved both the comprehension and metacomprehension accuracy of at-risk readers. The results suggest that poor metacomprehension accuracy can result from a failure to use appropriate cues for monitoring judgments, and that especially less-able readers need interventions that direct them to predictive cues for comprehension. Metacomprehension and Cue Use 3 Poor Metacomprehension Accuracy as a Result of Inappropriate Cue Use Learning from text is a standard adjunct to classroom instruction. Students are assigned reading for homework, where they are expected to study and understand textbook chapters or other texts. Models of self-regulated learning (e.g., Dunlosky & Thiede, 1998; Metcalfe, 2002; Nelson & Narens, 1990) suggest that metacognitive monitoring and regulation of study play an important role in such learning. Thiede, Anderson, and Therriault (2003) showed that monitoring accuracy (operationalized as the intra-individual correlation between metacomprehension judgments and test performance computed across texts) influenced decisions about which texts to restudy, which in turn affected learning from text. In particular, they showed that participants who more accurately monitored their comprehension made better decisions about which texts to reread than did participants who less accurately monitored their comprehension. That is, for a group with higher monitoring accuracy, participants chose to restudy primarily the texts that they did not understand. Their mean proportion correct on initial comprehension tests for the texts they selected to reread was .27 versus .78 for the texts they did not select to reread. By contrast, groups with lower monitoring accuracy showed less of a preference. The mean proportion correct on tests for the texts they selected to reread was .43 versus .53 for those they did not select to reread. The more effective regulation of study among the group with higher monitoring accuracy produced higher overall reading comprehension on subsequent tests for that group. Given that these results show that better comprehension monitoring accuracy can lead to better learning from text, it is important to find ways to improve comprehension monitoring accuracy, which has been called metacomprehension accuracy. It is highly problematic, then, that the usual level of metacomprehension accuracy is generally quite dismal, with correlations between predicted test performance and actual Metacomprehension and Cue Use 4 performance hovering around .27 (Maki, 1998a, Dunlosky & Lipko, 2007). Prior research has identified a number of constraints that prevent readers from engaging in accurate metacomprehension, but perhaps the most critical one is that readers generally are not basing their judgments on predictive cues for actual comprehension (Rawson, Dunlosky, & Thiede, 2000; Thiede, Wiley, Griffin & Redford, in press; Wiley, Griffin & Thiede, 2005). A great deal of research has been dedicated to identifying the cues that readers use to judge comprehension. This research has suggested that readers use such cues as domain familiarity or interest in the topic (Glenberg & Epstein, 1987; Glenberg, Sanocki, Epstein, & Morris, 1987; Maki & Serra, 1992), accessibility of information in memory (Baker & Dunlosky, 2006; Morris, 1990), ease of processing the text (Dunlosky & Rawson, 2005; Dunlosky, Rawson, & Hacker, 2002; Maki, Foley, Kajer, Thompson, & Willert, 1990; Rawson & Dunlosky, 2002), and global characteristics of texts such as length or difficulty (Weaver & Bryant, 1995). However, these cues may or may not lead to accurate judgments of test performance, depending on the nature of the test that is given. Some of these cues may produce good monitoring accuracy when tests are memory-based, but not when the tests require understanding of connections, or the generation or recognition of inferences based on the text. To understand the cues that may predict performance on these sorts of tests requires bridging theories of metacognitive monitoring with theories of comprehension (Rawson et al., 2000; Wiley et al., 2005; Weaver, 1990). Several successful interventions have been informed by such an approach, which has been called a situation model approach to metacomprehension (Griffin, Wiley & Thiede, 2008; Thiede et al., in press; Wiley et al., 2005). This approach is based on the comprehension framework of Kintsch (1994, 1998) which posits that a reader creates multiple representations of a text as he or she reads. For instance, the reader constructs a representation of the text at a Metacomprehension and Cue Use 5 surface level (e.g., the exact words), a textbase level (e.g., the meaning of sentences), and the situation-model level, where connections are made across units of the text as well as with prior knowledge. A well-constructed situation model integrates across the ideas contained in a text and allows the reader to form a causal model and inferences implied by the text. When tests of comprehension actually tap the situation model of a text (Kintsch, 1994; McNamara, Kintsch, Songer, & Kintsch, 1996; Wiley et al., 2005), metacomprehension accuracy should increase if readers use cues that tap the situation model level of representation to judge their comprehension. Furthermore, if readers are using cues other than those related to the situation model, monitoring attempts might be misdirected, which would result in poor metacomprehension accuracy. Support for the situation model approach has been found across a number of studies. Thiede and colleagues (i.e., Thiede & Anderson, 2003; Thiede, Anderson, & Therriault, 2003) were able to increase monitoring accuracy from .27 to .60 with the use of summarizing and keyword listing tasks that were performed prior to judgment. However, this improvement only occurred when the tasks were performed at a delay after reading and not when performed immediately. The authors explained this delay effect in terms of whether the generation task involved accessing STM or LTM representations of the texts. However, subsequent work by Thiede, Dunlosky, Griffin, and Wiley (2005) demonstrated more convincingly that these previous effects were due to performing a generative task that required accessing and employing one’s text representation after a delay. Readers are getting access to cues when they access their text representations and these cues are more predictive of comprehension test performance when accessed at a delay. This is true, even though the keyword listing task itself could be considered little more than a simple word recall task. Thiede et al. (2005) argued that the key factor is the Metacomprehension and Cue Use 6 level of representation being accessed. When performed immediately, these generation tasks can be performed using the highly accessible surface representation, but at a delay the situationmodel representation is more likely to be accessed due to the reduced accessibility of surface model. This interpretation is based on the work of Kintsch, Welsh, Schmalhofer and Zimny (1990; see also Fletcher & Chrysler, 1990; Schmalhofer & Glavanov, 1986), which has shown that access to surface information decays rapidly, whereas access to the situation model is more robust over time. As surface memory is less accessible at a delay, it is less likely to be used by readers as a basis for their comprehension judgments. Thus, the cues produced by the same task of recalling keywords varied in the degree of predictive validity when performed immediately versus at a delay, due the difference in the level of representation involved in performing the task. At this point, it is important to clarify that cues which differ in their validity may not always differ in terms of the general cue type they represent. Cues can be categorized into different broad types, such as superficial (e.g., familiarity, interest), memory-based (e.g. recallability), and comprehension-based (e.g., ability to self-explain). Cues are valid when they happen to reflect the level of representation being assessed at testing. Certain cue types (e.g., ability to self-explain) may have consistent ties to a certain level of representation (e.g., the situation model). However, some cue types, like ability to recall keywords, may reflect varying levels of representation, depending on contextual factors like the time elapsed since reading. Thus, our discussions about readers using more valid cues may sometimes involve switching from one cue type to another, but may sometimes involve the same general cue type becoming a more valid predictor of performance. Metacomprehension and Cue Use 7 Griffin et al. (2008) presented converging evidence for the situation-model approach by showing that directing readers toward their situation model via a “self explanation” instruction improved relative accuracy (r = .63). In this study, the self-explanation instruction prompted readers to explain the meaning and relevance of each part of the text to other parts and to the overall purpose of the text. Such explanation-based reading tasks have been shown to focus readers on their situation-model representations (Chi, 2000; Wiley & Voss, 1999). A further important point is that explanation occurred during reading and not at a delay. As no delay was involved, some alternative interpretations of the previously observed improvements in accuracy due to delayed generation effects (such as transfer-appropriate monitoring) are not viable explanations for the self-explanation effect. In all these prior studies, the presumption is that the manipulations are improving access to the situation model, or more specifically, improving access to cues related to the quality of the situation model, and as a result, the interventions shift readers from monitoring poor cues to better cues for predicting their own comprehension. However, previous studies offer only indirect evidence to support this presumption. In Experiment 1, the delayed summary paradigm of Thiede and Anderson (2003) is again employed, but in addition, readers are asked to report the cues they are using to judge comprehension, thus providing the first direct investigation of this issue. Another extension to Thiede and Anderson’s original study in the present investigation is in terms of the sample that was run. The original study did not explore whether the delayed summary intervention might be effective for readers of all ability levels. Previous work has suggested that less-able readers might have poorer metacognitive skills than more-able readers (Garner, 1987). However, no studies have specifically investigated the metacomprehension accuracy of at-risk college readers. Metacomprehension and Cue Use 8 Griffin et al (2008) recently showed that variability in reading comprehension skill among normal college readers was significantly related to metacomprehension accuracy. The present inclusion of at-risk readers allowed us to examine whether a robustly successful intervention like delayed generation serves to widen, narrow, or simply maintain the accuracy gap between more and less skilled readers. Also, if at-risk readers show both lower accuracy and reliance on more superficial judgments cues, then this would support the claim that cue validity is a critical factor in determining monitoring accuracy. Thus, a main goal of the current studies was to explore whether we might observe a relation between ability and metacomprehension accuracy when comparing a typical college reader sample to a sample that the university required to attend remedial reading classes. Assuming such an effect would be found, of interest is exploring the possible reasons for poor performance among the at-risk reader reading sample, and in turn, what instructional contexts might address those issues and increase metacomprehension accuracy among at-risk readers. Experiment 1 The primary purpose of this experiment was to further test the situation model approach to improving metacomprehension accuracy by evaluating whether the use of cues relevant to the situation model is associated with higher levels of accuracy. In this experiment, we replicate the procedures of Thiede and Anderson (2003) with both a typical college sample as well as a sample of college students who were required to enroll in remedial reading courses by the university. Students from both samples completed all three conditions: no summary, immediate summary and delayed summary. Each condition was run in a separate session, and order of conditions was counter-balanced. In addition, students were asked to report the basis for their judgments of comprehension. Metacomprehension and Cue Use 9 This is a more direct way of ascertaining cue use than has been used in previous research, which has relied on correlational data and the effects of targeted manipulations to infer shifts in the bases of metacomprehension judgments. Thus, in Experiment 1, the effects of different summary conditions were tested on both normal and at-risk college reader samples. And, the relation between cue use and metacomprehension accuracy was analyzed to explore possible explanations for poor accuracy, as well as potential differences between the two reader groups Method Participants. One hundred forty-two college students participated as a course requirement. Of the 142 who began the study; 15 (10.5%) failed to complete all three sessions and were dropped from the study. Although 127 students completed each of three sessions required for this experiment, 21 participants failed to respond to openended questions about cue use or had indeterminate gamma correlations, due to invariance in metacomprehension judgments; thus, only 106 had complete data. Of these, 32 were students recruited from a developmental reading course. These students (who had a mean ACT score of 14.2) were classified as at-risk readers by the university on the basis of their ACT scores (<18), and were required to enroll in the remedial reading course. In addition, 74 were students recruited from an introductory psychology course (who had a mean ACT score of 23.2) who were not required to take a remedial reading course. Although ACT score is not a pure test of reading ability, this was the measure used by the university to assign students to the remedial reading course, so it is used as the reading ability criterion for this study. The samples will be referred to as Metacomprehension and Cue Use 10 “typical college” and “at risk” readers in reflection of the manner in which they were selected. Although at-risk readers were actively recruited, the pool of students enrolled in the developmental reading course was far fewer than that enrolled in the psychology course; thus, it was not possible to obtain equal numbers of readers from the two groups in the study. All participants were treated in a manner consistent with the ethical standards of the American Psychological Association. Materials. The texts were adapted from ACT test preparation materials. They ranged in length from approximately 600 to 800 words, and had an average FleschKincaid readability score of 11.4. Three sets of five texts were constructed with a balance of topics from three general categories: natural science, social science, and humanities. The tests contained 10 multiple-choice items (with four alternatives) designed to assess comprehension (inference-making or application), rather than memory of details contained in the text (an example is presented in the Appendix). Design. This study utilized a mixed design with reading group being a betweensubjects variable and summarization condition being a within-subjects variable. That is, each participant completed each of summarization conditions in a separate session: nosummary, immediate summary, and delayed summary. Experimental sessions were separated on average by five days. A Latin-square design was used to counterbalance the order of tasks. Order was manipulated as a means of control, and was not expected to interact with the other variables of interest. A set of preliminary analyses confirmed that there were no significant order effects or interactions with outcome variables, all Fs < 1.9, p > .10. Hence, order is not considered in the main experimental analyses. Metacomprehension and Cue Use 11 Procedure. All participants were instructed that they would read texts on a computer screen, rate their comprehension for each text, and then answer test questions for each text. They were also instructed that they might be asked to write a summary for some of the texts. Finally, they were instructed that they would respond to some questions regarding the tasks in the experiment. Following instructions for the first session, the participants read a sample text and rated their comprehension of the text. The comprehension rating was prompted with the title of the text at the top of the computer screen and the question (as in Glenberg and Epstein, 1985), “How well do you think you understood the passage whose title is listed above? 1 (very poorly) to 7 (very well).” After rating their comprehension, participants answered a practice cue report question, which asked, “When you finish reading text material, how do you decide whether you have understood the passage? That is, when asked to “grade” your comprehension of that passage, what do you base your grade on so you can say, ‘I understood this passage well’ or ‘I read it, but I didn’t understand it’?” After typing their response to the practice question, they answered sample test questions. During the no-summary task, participants read five texts. After reading all texts, they rated their comprehension for each of the texts. (For comparison to other studies, this represents the standard delayed judgment condition, because judgments are made after all texts are read and not immediately after reading.) After rating their comprehension of the last text, participants responded to two open-ended questions. One question required a global response and asked, “You just rated your comprehension of five passages. What did you use to decide whether your comprehension over a passage Metacomprehension and Cue Use 12 was given a high rating or a low rating?” The computer then showed them the title of a passage they rated low and the title of a passage they rated high, and required a comparative response, “You gave a lower rating to your comprehension for the passage entitled (Title A) and a higher rating to the passage entitled (Title B). Describe the differences between the passages and your reading experience that made you give different ratings of comprehension.” The participants then answered the 10 multiplechoice questions for each text. During the immediate-summary task, participants read the first text displayed on the screen. They were then shown the title of the text and instructed to write a summary of that text. Once they finished writing the summary, they were presented with the next text. They read and immediately wrote a summary of each of the five texts. After writing the summary of the last text, participants rated their comprehension of each text and then answered the global and comparative cue-use questions. After typing their responses to the questions, they answered the 10 multiple-choice test questions for each text. During the delayed-summary task, participants read all five texts. They were then shown the title of the first text they had read and were instructed to write a summary of that text. When they were finished with this summary, they were presented with the next title and asked to write a summary of that text, and so on for all texts. After writing a summary of the last text, participants judged their comprehension of each text and answered the global and comparative cue-use questions. After typing their responses to the interview questions, participants answered the 10 multiple-choice test questions for each text. Metacomprehension and Cue Use 13 For all conditions, the texts were presented in a randomized order for each participant. Texts were rated for comprehension and tested in the same order as they were presented for reading. After answering the last multiple-choice test question in each task, participants were presented with the number of questions they had correctly answered over all five tests. That is, they received feedback regarding overall performance; they did not receive feedback regarding performance on a test for a particular text. They then responded to a closed-ended test-expectation question which asked, “Were you surprised at the score you got on the comprehension questions?” For the immediate-summary and delayed-summary conditions, the participants also responded to a closed-ended summaryuse question which asked, “Did you think about your summary when you made your rating for comprehension?” Coding. Responses to the open-ended questions provided self-report information on the different cues used to judge comprehension. For the global and comparative cue use questions, a research assistant, who was blind to the condition, compiled a list of 30 cues that participants reported using to judge comprehension. These cues were collapsed into five cue types: ability to explain meaning (e.g., “I gave it a high number if I thought I could explain the meaning of the story to another person.”), ability to recall or restate information about the text (e.g., “I based my rating on how well I could remember the ideas of the article.”), prior knowledge of a topic (e.g., “I gave it a high rating because I knew a lot about the topic.”), interest in the topic of the text (e.g., “I gave it a low rating because I think Beethoven is boring.”), and use of features of the text including difficulty, ease of processing, readability, length and specific vocabulary (e.g., “I gave it a low rating because it was long and hard to read.”). Readers’ responses were not restricted and Metacomprehension and Cue Use 14 could represent more than one of these 5 cue types. A second research assistant coded approximately 30% of the responses. The inter-rater reliability was quite high (kappa = .93). Actual cue use is not directly observable to researchers, so any measure of cue use will have potential limitations. Self-reports of cognitive processes are a general concern (as described by Nisbett & Wilson, 1977). Readers may not have access to or awareness of the cues they rely upon, and they could just randomly report any plausible cue types that come to mind. Alternatively, readers could be generally biased towards reporting cues that seem more sophisticated than what they actually using. Such measurement errors are largely a problem for conclusions about the absolute levels on univariate distributions. However, these potential measurement problems are manifested as nullresults in multivariate analyses, so the self-reports can be validated via their systematic relationships with other measures (see Erricsson & Simon, 1980). The primary focus of the present studies will be the multivariate relationship between reported cue use and metacomprehension accuracy, and how these covary across different reading levels and experimental conditions. For the test-expectation question, participants overwhelmingly (over 90%) responded that the test was what they expected and that they were not surprised by the kind of test questions they had received. As a result, this question yielded no useful information for the purposes of this study and is not discussed further. Results The first step in analysis was to see whether the effects in the delayed summary condition replicated the earlier work. Metacomprehension accuracy was operationalized Metacomprehension and Cue Use 15 as the gamma correlation between comprehension ratings and performance on a test of reading comprehension computed across texts following the procedure of Thiede and Anderson (2003; see Nelson, 1984, for a rationale for using gamma). Thus, before computing metacomprehension accuracy, descriptive analyses on judgments and test performance are presented. Then metacomprehension accuracy is considered. This is followed by an analysis of which cues were reported to be used as a basis for metacomprehension judgments and how cue use related to accuracy. Metacomprehension judgments. The median of metacomprehension judgments across the five texts was computed for each participant. The median is the recommended measure of central tendency for small sets of scores where extreme scores could affect the mean (Gravetter & Wallnau, 1999). The mean of the medians was computed across participants by condition. A 2 (reading group) x 3 (summary condition) ANOVA showed there was a main effect for reading group, F (1, 104) = 8.14, MSE = 1.03, p < .01, η= .07. As seen in Table 1, judgments were higher for typical readers than at-risk readers. There was also a main effect for summary condition, F (2, 208) = 4.07, MSE = .75, p < .02, η = .04, with follow-up tests indicating higher judgments in the no summary condition, compared to the other two. The interaction was marginal, F (2, 208) = 2.26, MSE = .75, p < .10, η = .02. Importantly, similar variance in judgments was seen across conditions and reading groups, and there were no ceiling or floor effects. Test Performance. The median proportion of correct test responses across the five texts was computed for each participant. The mean of the medians (presented in Table 1) was then computed across participants within each condition. Metacomprehension and Cue Use 16 A 2 (reading group) x 3 (summary condition) ANOVA revealed a significant main effect for reading group, F (1, 104) = 46.2, MSE = .02, p < .0001, η = .31. The main effect for summary condition was marginal, F (2, 208) = 2.24, MSE = .01, p < .10, η = .02. The interaction was significant, F (2, 208) = 3.40, MSE = .01, p < .04, η = .03. Follow-up tests revealed that typical college readers performed better on these ACT-type passages than the at-risk readers, as would be expected since the samples were selected based on actual ACT scores. Moreover, the interaction was due to the typical readers doing worse in the immediate summary condition than in the other two conditions. Importantly, both groups showed similar variance in their performance and there were no ceiling or floor effects. Metacomprehension Accuracy. Metacomprehension accuracy was operationalized as the gamma correlation between comprehension ratings and test performance across a set of texts. In this study, three intra-individual correlations were computed for each participant, one for each summarization condition. The mean gamma correlation was then computed across participants for each condition. As seen in Figure 1, metacomprehension accuracy differed significantly across conditions, F(2, 208) = 19.3, MSE = .22, p < .001, η = .16. Consistent with the findings of Thiede and Anderson (2003), follow-up tests found that the delayed-summary condition increased accuracy over the immediatesummary and no-summary conditions, which did not differ. A main effect of reading group was also found, as metacomprehension accuracy was greater for typical readers than for at-risk readers, F(1, 104) = 4.97, MSE = .09, p = .03, η = .05. Summary condition did not interact with reading group, F < 1. The lack of an interaction indicates that the delayed summarization instruction was not a strong Metacomprehension and Cue Use 17 enough intervention to equate the accuracy of the two reading groups. As a result of the two main effects, typical readers reached the highest level of accuracy in the delayedsummary condition (around .60) and remained more accurate than the at-risk readers who only achieved accuracy (around .45) even in the delayed-summary condition. Cues used to Judge Comprehension The responses to the global and comparative cue use prompts revealed largely similar distributions of cue use across conditions and reading groups. Because of their extreme similarity, only data for the global prompt are presented here. As mentioned above, comments were originally sorted into five categories. The frequency of responses from this initial coding is presented in Table 2. Note that in this table, participants can contribute to more than one cue type in each condition. In order to create an exclusive coding system for analyses, readers were classified into one of four cue-use profiles: surface, reader, memory, or comprehension. Readers who reported using any cues related to the qualities of the text itself were classified as fitting the surface profile, regardless of any other cues that were reported. Those who reported relying on their own ability to understand or explain the text, but not surface cues, were classified as using comprehension-based cues. Readers who referred to their ability to recall the text, but not comprehensionor surface-based cues were classified as using memory-based cues. Finally, readers who reported relying on judgments about their own level of familiarity with or interest in the topic, without mentioning the text’s surface features, memory-based cues, or comprehension-based cues were classified as relying on reader-based cues. Because of low numbers of observations in the prior Metacomprehension and Cue Use 18 knowledge and interest categories, these two cue sets were collapsed into a single “reader characteristics” category. The decision to assign all readers using any surface cues to a surface profile was driven by observation of the data in the neutral condition. First, looking at the readers who reported only surface cues, we observed very low gammas among this group. The participants who reported using only surface cues had a mean gamma of -.03, while participants who reported only reader-based cues had a mean gamma of .19, those reporting only memory-based cues had a mean gamma of .20, and those reporting only comprehension-based cues had a mean gamma of .71. Next we examined the performance of participants who reported a combination of cue types from multiple categories. For both surface/reader combinations (-.13), and surface/comprehension combinations (-.33), the gammas were quite dissimilar from those for readers who used exclusively reader and comprehension profiles. (Although for readers in the memory-based profile, the combination with surface cues if anything improved performance, .35). Given that in most cases, reporting use of any surface cues made readers appear more similar to those who reported only surface cues, and the point of this analysis was to attempt to characterize the behaviors that related to accurate or inaccurate metacomprehension, we elected to collapse all the combinations that included surface cues into the surface profile category. By the same logic, we examined the effects of reader cues in combination with memory (.36) and comprehension-based (.84) cues. In both cases, behaviors were consistent with performance in the pure “memorybased” and “comprehension-based” conditions. Thus in these cases the combinations Metacomprehension and Cue Use 19 were collapsed into the higher-order categories. By this process, we determined our classification of cue use profiles to be used for all conditions. The above classification was done three times for each individual, once for each summary condition based on their comments at the end of each condition. The overall frequency of readers falling into each profile type by reading group and summary condition are presented in Figure 2. First note that, overall, comprehension-based profiles were the least common, while memory-based profiles were the most common (i.e. cue use related to the ability to recall information from the text). Second, note that almost half of at-risk readers had a surface-cue profile, whereas typical readers were most likely to fall into the memory-based profile. Further, the distribution of profiles across conditions changed especially for the typical readers – who focused less on reader characteristics, and more on the quality of their ability to recall a text, when they made judgments in the delayed-summary condition. Pairwise Wilcoxon tests revealed no differences in distributions across summary conditions for at-risk readers (Zs < .57), whereas the distribution in the delayedsummary condition was different than the distribution for the other two conditions (Z=2.45 and Z=2.83, ps <.01) among the typical readers, with no difference between immediate-summary and no-summary conditions (Z < .54). Relation between cue use profile and metacomprehension accuracy. Several analyses were performed to explore the effect of summary condition on metacomprehension accuracy and whether cue use was related to accuracy. First, within each of the summarization conditions, between-subjects analyses were conducted to examine differences in metacomprehension accuracy due to cue use profiles. Next, Metacomprehension and Cue Use 20 within-subjects analyses were performed on subsets of participants who fit the same profile across conditions. Finally, a best-cue analysis was used to create a stable withinsubjects variable related to cue use, so that a fully within-subjects model could be tested. Within Summary Condition, Between Subjects Analysis. The overall patterns of metacomprehension accuracy for cue use and summary condition are presented in Figure 3. Note that in this figure, each participant has a monitoring accuracy score for each summary condition, but the cue use profile that an individual is assigned to can change across conditions. Thus, to analyze these data, we performed a separate ANOVA for each summary condition. For the no-summary condition, a 2 (reading group) x 4 (cue use profile) ANOVA revealed a main effect for cue use profile, F(3,101)=2.70, MSE=.30, p<.05, η = .08, but no effect for reading group, F < 1. (The interaction is not reported due to a lack of data in the at-risk reader/comprehension cell.) Follow-up tests revealed that monitoring accuracy was significantly worse for readers who fit a surface cue profile, and accuracy was significantly better for readers who fit a comprehension-based profile, than for other profiles. Accuracy for reader-based and memory-based profiles did not differ. For the immediate-summary condition, a 2 (reading group) x 4 (cue use profile) ANOVA revealed a main effect for cue use profile, F(3,101)=3.02, MSE=.22, p<.03, η = .08, but no effect for reading group, F < 1. (The interaction is not reported due to a lack of data in the at-risk reader/comprehension cell.) Follow-up tests revealed that monitoring accuracy was significantly better for readers who fit a comprehension-based profile than for all other profiles. Accuracy for the remaining profiles did not differ. Metacomprehension and Cue Use 21 For the delayed-summary condition, a 2 (reading group) x 4 (cue use profile) ANOVA revealed a main effect for cue use profile, F(3,98)=11.9, MSE=.11, p<.0001, η = .27, but no effect for reading group and no interaction, Fs < 1. Follow-up tests revealed that monitoring accuracy was significantly worse for readers who fit a surface-cue profile, and accuracy was significantly better for readers who fit a comprehension-based profile, than for other profiles. Accuracy for reader-based and memory-based profiles did not differ. Consistent Cue Use Profile, Within-Subjects Analysis. The analyses within each summary condition revealed that readers who rely on cues based on surface features of the text had lower metacomprehension accuracy, and those who rely on comprehensionbased cues had greater accuracy. However, an interesting pattern can also be seen if one looks across summary conditions, as it appears that the utility of using memory-based and reader-based cues changes, and that such cues are only predictive in the delayedsummary condition. To test that increases in predictive accuracy are due to the summary condition, and not due to particular individuals who only fall into a memory-based profile in one condition but not the other, we computed the average gammas for only the subset of participants who fell into the memory-based profile in both immediateand delayedsummary conditions (N=30). For these participants, gammas were significantly higher in the delayed-summary condition (.67, SE .06) than in the immediate-summary condition (.25, SE .07), t(29)= 4.9, p<.001. Thus, relying on memory-based cues as a basis for predictive judgments can be a particularly effective strategy, but this is only the case when these judgments follow summaries that are generated at a delay. Metacomprehension and Cue Use 22 When the same analysis is performed for the reader-based profile, only 4 participants fell into this category in both the immediateand delayed-summary conditions, and their means did not differ (immediate: M = .45, SE .20; delayed: M = .44, SE .28, t<1). Thus the change between relatively low metacomprehension accuracy in the no-summary and immediate-summary conditions, and high accuracy in the delayedsummary condition, is due to movement of individuals into different profile types across conditions. Best Cue Reported, Within Subjects Analysis. In order to compare across summary conditions in a more powerful way using the full sample, a further analysis assigned individuals to a single cue basis category, as a function of the highest quality cue that was used in any of the three summary conditions. The order of cue quality was based on theoretical premises that metacomprehension cues that assess the quality of understanding of the situation model level representation will be the most valid predictors of performance on a test of comprehension (Griffin et al., 2008; Thiede et al., in press; Wiley et al., 2005). Therefore, reports of cues related to the quality of understanding or ability to explain the content of the passage were rated as highest in quality (i.e. the comprehension-based cue category described above). The next set of cues in terms of quality were the memory-based cues that again referenced a readers’ reflection on their own representation, but commented specifically on the ability to remember the text (as opposed to the ability to understand it). This level of monitoring activity corresponds theoretically to assessing the quality of the textbase, which can be a predictor of performance on comprehension tests in some cases, but this is Metacomprehension and Cue Use 23 not necessarily true and may be predictive to a lesser degree than situation-model level judgments (Wiley et al., 2005). The final two sets of cues were classed as lower in quality, because neither required readers to reflect on their own representation of the texts. The third class of comments was those that referred to predictions based in reader characteristics of personal interest or familiarity with the content of the texts. These cues can be predictive–having no interest in a test may accurately predict very low performance on a subsequent test due to a lack of motivation–but importantly they do not require reflection on or access to one’s own internal representation of the text. These comments instead refer to a quality of the reader, so they are self-assessments, but they may not necessarily relate to the comprehension of a particular text. The lowest class of cues was those that referred to qualities of the text itself– mainly the readability of text, the difficulty of the vocabulary used, and length of the passage. Again, these cues can be predictive of test performance, but they are heuristic approaches that do not require reflection on internal representations. As with all heuristics, they may lead to predictive judgments in some cases, but especially when comprehension performance is being predicted, they may be misleading. Thus, each participant was assigned to a single level of cue use based on the highest quality cue that was ever reported by the individual in any condition. This single measure of the best cue reported allowed for comparisons across summary instructions because with this coding individuals do not contribute to different categories across conditions. Therefore this approach provides an additional way to assess the effects of delayed summaries on metacognitive accuracy. Metacomprehension and Cue Use 24 Best Cue Reported as a function of Reading Group. Splitting the cue quality into heuristic (surface, reader) and monitoring (memory-based, comprehension-based) categories, we found that monitoring cues were more frequently used by typical college readers than for at-risk readers (87.1% vs. 66.6%), while heuristic cues were more frequently used by at-risk college readers than typical readers (33.3% vs. 12.9%).. This resulted in a significant chi-square (1)=6.27, p< .01, showing that distribution across the two best cue categories differed by reading group. Metacognitive Accuracy by Best Cue Reported. Figure 4 presents the average metacomprehension accuracy for each Best Cue group as a function of summary condition. A 3 (Summary Condition) x 4 (Best Cue group) ANOVA revealed significant effects for both summary condition, F(2, 204) = 12.3, MSE=.21, p<.0001, η = .11, and best cue group ((3, 102) = 7.03, MSE=.08, p<.001, η = .17). (There was no main effect for Reading Group, F<1, once cue use was included, so it was not included in the model.) The interaction between Summary Condition and Best Cue group was also significant, F (6, 204) = 2.57, MSE=.21, p<.02, η = .07. Follow-up tests for the summary condition effect revealed that metacomprehension accuracy in the delayed-summary condition was better than in the immediate-summary condition, which in turn was better than in the no-summary condition. Follow-up tests for the best cue effect revealed that use of comprehension-based cues led to better accuracy than all other cues. Use of memory-based cues was significantly worse than comprehension-based cues, but significantly better than surface Metacomprehension and Cue Use 25 or reader-based cues. Accuracy among those reporting use of reader and surface cues did not differ. To follow up the significant interaction between summary condition and best cue group, we tested for the presence of a main effect for summary condition within each cue condition. Within cue condition ANOVAs revealed that accuracy of readers whose best cue was in the surface or reader categories did not change as a function of summary condition. However, accuracy for readers whose best cue was memory-based did improve specifically in the delayed-summary condition. Further, accuracy for readers whose best cue was comprehension-based improved in the immediate-summary condition over the no-summary condition, and improved again with the delayed-summary condition over the immediate-summary condition, thus resulting in the highest level of metacomprehension accuracy in our sample. Summary from Cue Analyses. The findings from the best cue analysis converge with other cue-use analyses. Comprehension-based cues were the best predictors of performance on comprehension tests, but were rarely used. Memory-based cues were able to lead to valid predictions of test performance, but interestingly, only in the delayed-summary condition. This shift in the validity of memory-based cues is consistent with the explanation of the delayed summary effect as being a function of the changes in memory for text that occur after a delay (Thiede et al., 2005). Over time, memory for surface information fades while situation-model level information remains. Thus, when readers base their cues on their ability to remember a text, which become apparent during a summarization task, their judgments will be more Metacomprehension and Cue Use 26 predictive of comprehension test performance as long as some time passes after reading but before attempting to summarize. The present data provide support for this account. Finally, judgments based on surface characteristics of the text were the least predictive of performance on comprehension tests. And, in general, at-risk readers were more likely to use surface and reader cues, and less likely than typical readers to engage in metacognitive processes of reflecting on either their own level of understanding or their ability to remember texts in order to generate their judgments. This finding may perhaps be due to resource limitations that make the process of constructing a representation of text, and reflecting on it, too demanding (Griffin et al., 2008). Thus, atrisk readers may have been forced to resort to heuristic approaches to guide their judgment process. This hypothesis led to the formulation of a specific goal for the second experiment: to provide a context for at-risk readers that may give them direct access to valid cues for judgment and, at the same time, might allow them to reflect on the quality of their representations of texts. Use of Cues based in Summary Writing Experience. All of the above cue use analyses used the responses to the global question as a basis for determining the kinds of cues that readers were using. In addition, a final closed-ended summary-use question asked participants directly whether they thought about their summaries while making comprehension judgments. A large proportion of readers endorsed using this cue. Using information gained from the experience of writing a summary would be seen as an effective cue for judging comprehension. When readers have difficulty summarizing a text, this should alert them that their understanding is poor (Dunlosky & Rawson, 2005; Thiede & Anderson, 2003). In the immediate-summary condition, the proportion of Metacomprehension and Cue Use 27 typical readers who reported thinking about their summaries (66%) did not differ from that of at-risk readers (61%), (1) = . 01. However, in the delayed-summary condition, a greater proportion of typical college readers reported thinking about their summaries as they made their judgments (78%) than did at-risk college readers (44%), (1) = 6.98. Further, a 2 (reading group) x 2 (used summaries versus did not) ANOVA revealed that reported use of summaries affected accuracy in the delayed-summary condition. The main effect for summary use was significant, F(1, 102) =13.8. Neither the main effect for reading group nor the interaction were significant, Fs < 1. For both reading groups, accuracy was greater for those who reported using summaries as a cue for judging comprehension (Typical Readers: M = .73, SE .05; At-Risk Readers: M = .70, SE .11) than for those who did not (Typical Readers: M = .37, SE .09; At-Risk Readers: M = .44, SE .10). A 2 (reading group) x 2 (used summaries versus did not) ANOVA revealed that reported use of summaries did not affect accuracy in the immediate-summary condition. Neither main effect nor the interaction was significant, Fs < 1. These data provide additional evidence that the cues provided by the experience of generating a summary were more predictive of comprehension performance in the delayed-summary condition. They also converge with previous analyses showing that metacomprehension accuracy varies as a function of the cues that are used as a basis for comprehension judgments. At-risk readers who reported using summaries as a basis for their comprehension judgments were just as accurate as the typical college readers. Discussion Metacomprehension and Cue Use 28 Several important findings emerged from this study. First, the effectiveness of a delayed summary instruction was replicated in typical college readers and extended to a population of at-risk readers. Although the intervention did not close the gap between ability levels, it did improve metacomprehension accuracy overall. Second, based on self-reports of cues used as the basis for comprehension judgments, it appears that the benefits of the delayed-summary condition are indeed because it makes judgments based on memory-related cues more valid. As memory-based cues are the default basis for judgments for many readers, the delayed-summary condition improves metacomprehension accuracy by putting readers in a context where memory-based cues are predictive of comprehension performance. Across conditions, at-risk readers have less accurate judgments and were more likely to report using surface type cues when making those judgments. However, when at-risk readers did report using valid cues as the basis for their judgments (thinking about their ability to generate a summary), or not using poorer cues, then they were just as accurate as typical college readers. The fact that the reported cues can account for when at-risk and typical readers differ in monitoring accuracy lends support to the validity of this self-report measure of cue use. These findings are also consistent with the cue-utilization perspective on monitoring accuracy (Koriat, 1997). Besides the increased validity of recall cues in the delay condition, there was evidence that the different conditions had some effect on the type of cues readers used. Especially in the delayed-summary condition, metacomprehension accuracy improved when readers used valid cues such as the ability to generate a summary or explanation of a text as the basis for their judgments. On the other hand, focusing on simple surface and Metacomprehension and Cue Use 29 reader cues led to poorer accuracy. Importantly, this is the first study to attempt to provide direct evidence of the kinds of cues that readers use to judge comprehension. Perhaps most striking is how few participants spontaneously reported using cues that would be highly diagnostic of the quality of their situation models, i.e. the ability to explain the text. Only 11 readers mentioned this as a basis for their judgments. As disturbing is how many participants spontaneously reported surface features of the texts as the basis of their judgments of comprehension. This speaks to the need to give students a better understanding of what it means to comprehend expository text, so that they might base their comprehension judgments on more predictive cues (Wiley et al., 2005). The differences in metacomprehension accuracy and cue use as a function of reading proficiency highlight the need to explore additional interventions. Although some at-risk readers were able to perform as well as typical readers, this depended on them selecting valid cues for their judgments. There were still a large number of at-risk readers who were focused on incorrect cues for comprehension. Thus, Experiment 2 explored an intervention that explicitly directed less-able readers toward appropriate cues for judging their comprehension. Experiment 2 The results of Experiment 1 suggest that metacomprehension accuracy is influenced by the cues participants use to judge comprehension and that the validity of cues changed from one situation to another. Moreover, these results suggest that metacomprehension accuracy for many at-risk readers is compromised by the use of inappropriate cues (based on surface features of a text). In this experiment, we attempted Metacomprehension and Cue Use 30 to change the cues used by less-able readers to judge comprehension. In particular, we attempted to direct their attention to cues related to the situation model of texts by instructing them to construct concept maps as they read the texts. A concept map is a graphic representation of the underlying structure of the meaning of a text. Constructing concept maps can be an effective organizational strategy, which helps readers formulate the connections among concepts in a text (Weinstein & Mayer, 1986). Concept mapping was chosen as an intervention as it has been suggested that such an approach may be particularly helpful and appropriate for less-able readers (Stensvold & Wilson, 1990; for a review and meta-analysis on effectiveness of concept maps with low-ability learners, see Nesbit and Adesope, 2006). Concept mapping shares many similarities with argumentation and self-explanation tasks (Weinstein & Meyer, 1986), but because it employs the construction of external, visual representations while readers have access to the texts, it may place fewer demands on the reader than other explanation tasks. Instructing at-risk readers to construct a concept map of a text during reading should not only help them identify important connections, and therefore help them construct a situation model for a text, but it should also increase the salience of the quality of that situation model level representation, which they can then use to judge their comprehension of a text. Thus, we hypothesize that metacomprehension accuracy will improve when at-risk readers construct concept maps during reading (versus when they do not). Method Participants. Twenty-one students enrolled in a developmental reading course participated in the experiment as part of the course requirements (none of these students Metacomprehension and Cue Use 31 participated in Experiment 1). As in Experiment 1, all of these students had ACT scores less that 18 (M = 12.2, SE = .54) which required their enrollment in the remedial reading course. (All participants were treated in a manner consistent with the ethical standards of the American Psychological Association. Materials. The texts were adapted from on-line materials offered as supplementary readings for a developmental reading textbook. They ranged in length from approximately 250 to 350 words, and had an average Flesch-Kincaid readability score of 10. We constructed three sets of five texts with a balance of topics from three general categories: natural science, social science, and humanities. The sets of texts were randomly assigned across condition for each participant. The tests contained five multiple-choice items designed to assess comprehension (inference making or application), rather than memory of details contained in the text, an example of the texts and tests are in the Appendix. Design. This study utilized a within-subjects design. Each participant first completed immediate-judgment and delayed-judgment conditions, with the order of these conditions counterbalanced. These conditions were completed on separate days with one week between sessions. The order of conditions did not affect any of the outcome variables, ts < 1.3; therefore the order of these conditions was not included in subsequent analyses. All students then completed concept map training and ran in the concept map condition. Procedure. All participants were instructed that they would read texts on a computer, judge how well they understood each text, and then take a test for each text. In the delayedjudgment task (the control condition used in Experiment 1 which is the standard in the metacomprehension literature, Maki, 1998b), participants read all five texts, then made metacomprehension judgments for the texts in a block, and then answered test questions for the Metacomprehension and Cue Use 32 texts in a block. The prompt for metacomprehension judgments in this experiment was the same as in Experiment 1. A second comparison condition was also run using immediate judgments. In the immediate-judgment condition, participants read a text and judged their comprehension of the text immediately after reading. After making their judgment, participants answered five multiplechoice questions on the text. They completed this procedure for all five texts. Maki (1998a) has shown that an immediate judgment condition can produce higher levels of metacomprehension accuracy than the standard delayed-judgment condition. But the main reason for including this condition was because the concept map condition used immediate judgments following the construction of each concept map, and this control condition was needed to provide a wellmatched comparison. After completing these two conditions, participants received eight 50-minute class periods of instruction and practice constructing concept maps. They then completed the concept map condition, and received a new set of texts in a concept map condition on the day after the final period of instruction. In the concept-map condition, for each text in the set, participants constructed a concept map while reading. After reading and constructing a concept map, participants made their metacomprehension judgment (without access to their concept maps), and then answered five multiple-choice questions on the text. Participants read and constructed concept maps, judged comprehension, and answered test questions for all five texts. Thus, the procedure was like the immediate-judgment condition except that participants constructed concept maps while reading. Metacomprehension and Cue Use 33 Results and Discussion Test performance and comprehension ratings. As metacomprehension accuracy describes the relations between comprehension ratings and performance on a test of reading comprehension, descriptive statistics of these variables are reported first. The median proportion of correct test responses and metacomprehension judgments across the five texts was computed for each participant. The mean of the medians was then computed across participants within each condition. Test performance differed across conditions, F(2, 40) = 15.3, MSE = .44, p < .001, η = .43 (see Table 1). Follow-up tests showed that test performance was greater for the concept map condition than the immediate-judgment condition, t(20) = 4.0, p < .001; or the delayed-judgment condition, t(20) = 5.6, p < .001. Thus, constructing concept maps while reading improved comprehension, which is consistent with the literature. As seen in Table 1, the magnitude of metacomprehension judgments did not differ across conditions, F(2, 40) < 1. Metacomprehension accuracy. Metacomprehension was again operationalized as the Goodman-Kruskal gamma correlation between metacomprehension judgments and performance on a test of reading comprehension computed across texts. Three gamma correlations were computed for each participant, one for each condition. The mean intra-individual correlation was then computed across participants for each condition. One participant in each condition had an indeterminate gamma correlation due to invariance in judgments. Metacomprehension accuracy differed across conditions, F(2, 36) = 3.6, MSE = .22, p < .05, η = .17 (see Figure 5). Follow-up tests showed that metacomprehension accuracy was greater for the concept map condition than for the immediate-judgment/test condition, t(18) = 2.3, p < .05; or the delayed-judgment/test Metacomprehension and Cue Use 34 condition, t(19) = 2.4, p < .05. No differences were found between the two control conditions, suggesting that a delay before judging neither helped nor hurt metacomprehension accuracy. The above finding suggests that participants used something about their experience with their concept maps as a basis for their metacomprehension judgments. To evaluate this possibility, we first coded the number of appropriate connections made between concepts within each concept map as a metric of the quality of the representation. A second research assistant then scored the concept maps of 6 participants (approximately 30% of the responses) and scored the number of connections. The inter-rater reliability on coding these connections was quite high (kappa = .94). For each participant, we then computed a gamma correlation between the number of connections and metacomprehension judgments across the texts. The mean intra-individual correlation between the number of connections and metacomprehension judgments was .32 (SEM = .13), which is significantly different from zero, t(19) = 2.5, p < .05. This suggests that participants used the quality of their concept maps as basis for their judgments of comprehension. For this to help explain the improved metacomprehension accuracy, concept maps would need to be predictive of test performance. To evaluate this possibility, we computed a gamma correlation between the number of connections and test performance across the texts. The mean intra-individual correlation between the number of connections and test performance was .38 (SEM = .11), which is significantly different from zero, t(20) = 3.4, p < .01. Thus, the number of connections included in concept maps was predictive of test performance. The concept maps thus served as a mechanism that made the quality of the situation model for each text salient to the reader, and therefore provided a predictive basis for metacomprehension judgments. General Discussion Metacomprehension and Cue Use 35 The present studies offer several important findings. The first is that self-reports of cue use indicated that poorer metacomprehension accuracy can be seen as a function of inappropriate cue use. That is, when readers do use appropriate cues to judge comprehension, they make accurate judgments. The second important finding is at-risk readers tended use more inappropriate cues. This finding led to the suggestion that low-ability readers might need interventions that aid them in selecting valid cues. In response to this issue, the final important result was that introducing a concept mapping intervention to a sample of at-risk readers improved both their comprehension and their metacomprehension as they attempted to learn from texts. This suggests that less-able readers benefited from a task that guided their learning of expository text, by teaching them to attend to the connections that could be made within each text. Consistent with previous research (e.g., Nesbit & Adesope, 2006; Stensvold & Wilson, 1990), we found that concept maps were effective learning tools for less-able readers. In the current study, we go beyond previous results to show that such an intervention improved metacomprehension accuracy in less-able readers as well as learning outcomes. Using a concept map intervention, less-able readers were able to reach a level of metacomprehension accuracy (around .60) that was comparable to the best levels that have been achieved in the literature. The present studies provide additional support for the situation model approach to improving metacomprehension. One important property of concept-mapping is that it gets readers to attend to the quality of the situation model that they are constructing. The fact that improvements in metacomprehension accuracy were seen as a result of this activity is consistent with other studies that have improved metacomprehension via interventions that make the quality of the situation model salient to readers, including generating keywords or summaries at a delay (Thiede & Anderson, 2003; Thiede et al., 2005), or generating self-explanations (Griffin et al., Metacomprehension and Cue Use 36 2008). However, all of these studies have been conducted with skilled or typical reader populations. The present study extends previous work into a new population that sorely needs support in expository text comprehension. In Experiment 1 it was especially at-risk readers who used inappropriate cues to judge their own comprehension. The positive effects of concept mapping tasks observed here, suggest it may be a promising alternative to self-explanation that may be especially appropriate for younger or less-able readers. Such readers may not be able to handle the additional load imposed by monitoring, self-explanation or delayed-generation activities without the supportive assistance of the external representations that concept mapping provides. Consistent with previous research that argues for the importance of using the situation model as a basis for comprehension judgments, we found that metacomprehension accuracy was greater for participants who reported using their ability to explain the meaning of texts as a cue for judging comprehension (versus those who did not report using this cue). Moreover, given that constructing concept maps may help less-able readers formulate a situation model for a text and attend to their situation model when judging comprehension, the findings from Experiment 2 provide additional evidence that getting readers to focus on their situation model during reading will improve metacomprehension accuracy. Further research now needs to be done to illustrate how and when students can translate monitoring accuracy into effective regulation of their own study behaviors, including making better choices of what to read and reread while studying, which in turn, will ultimately improve learning from expository text. Metacomprehension and Cue Use 37